{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0f0b9afa",
   "metadata": {},
   "source": [
    "### 🚀 WaterCrawl × FLARE — the perfect duo for RAG playgrounds!\n",
    "\n",
    "Welcome to this **step‑by‑step Jupyter Notebook** where we:\n",
    "\n",
    "1. 🕷 **Crawl & clean** any website with **WaterCrawl** – turning raw HTML into markdown/JSON that’s ready for embeddings.   \n",
    "2. 🔍 **Retrieve on‑the‑fly** with **FLARE (Forward‑Looking Active REtrieval)** – an “always‑be‑fact‑checking” wrapper that pulls extra docs *only* when the LLM shows low confidence.  \n",
    "3. 🛠 **Tie it all together** with **LangChain**, **Tavily Search API** & a few helper utils so you can remix the pipeline to your heart’s content.\n",
    "\n",
    "---\n",
    "\n",
    "#### What’s inside?\n",
    "\n",
    "| 🔧 Component | 💡 Why we’re using it |\n",
    "|--------------|----------------------|\n",
    "| **WaterCrawl** | Point‑&‑shoot crawling with sitemap visualizer, duplicate detection, and markdown/JSON exports – perfect for vector DB ingestion. :contentReference[oaicite:0]{index=0} |\n",
    "| **LangChain** | Glue layer that lets us chain the crawl → embed → FLARE retrieval steps with a few lines of code. |\n",
    "| **Tavily Search API** | Fast, inexpensive web search that slots into `TavilyRetriever`; great complement to your own crawled corpora. |\n",
    "| **FLARE** | Re‑checks the model’s “next sentence” for shaky tokens; if confidence is low, it auto‑generates a smart query and fetches fresh docs before writing. :contentReference[oaicite:1]{index=1} |\n",
    "\n",
    "---\n",
    "\n",
    "#### Notebook flow 🗺️\n",
    "\n",
    "1. **Setup**: grab your API keys from https://watercrawl.dev/, spin up your own `watercrawl` from: https://github.com/watercrawl/watercrawl. To run WaterCrawl API you need to install the Python SDK, which we will do in the following steps\n",
    "2. **FLARE chain**: initialize `FlareChain(llm_answer, llm_question, retriever)` with **Tavily** + your newly‑minted vector store.  \n",
    "3. **Ask away!**: watch FLARE pause, retrieve, and resume writing—as many times as needed—to give rock‑solid answers.  \n",
    "4. **Extras**: show off the visual sitemap PNG WaterCrawl generated and link each node to its vector IDs.  \n",
    "\n",
    "---\n",
    "\n",
    "### Why you’ll ❤️ this combo\n",
    "\n",
    "- **Less hallucination, more citation**: WaterCrawl hands FLARE pristine, source‑mapped text, so every sentence can be traced back to a URL.  \n",
    "- **Pay only for what you need**: FLARE calls Tavily *selectively*, not on every token—so your search bill stays tiny.  \n",
    "- **Drop‑in for any stack**: swap Tavily for your own BM25/Elastic/Weaviate retriever, or point WaterCrawl at authenticated intranet sites.  \n",
    "- **Open‑source all the way**: MIT‑style licences on both projects mean you can fork, tweak, and ship to prod. :contentReference[oaicite:2]{index=2}\n",
    "\n",
    "> **Tip:** if you’re new to WaterCrawl, follow: https://github.com/watercrawl/watercrawl?tab=readme-ov-file#-quick-start  hit `http://localhost` after `docker compose up -d` and explore the Playground UI—selector testing & screenshot capture included! 🎨\n",
    "\n",
    "---\n",
    "\n",
    "Ready? Let’s spin up containers and start crawling! 🏁\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7e41fe0",
   "metadata": {},
   "source": [
    "##### ➡️ **Lets install 📦all the dependencies:** \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0f902f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade pip\n",
    "!pip install  langchain-community langchain-core langchain-openai notebook watercrawl-py tavily-python\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "212bbe7f",
   "metadata": {},
   "source": [
    "### ➡️ 🔑 **API keys you’ll need (grab these first!)** \n",
    "\n",
    "| Service | What it’s for | Where to generate |\n",
    "|---------|---------------|-------------------|\n",
    "| **WaterCrawl** | Auth for crawling endpoints | <https://app.watercrawl.dev/dashboard/api-keys> |\n",
    "| **OpenAI** | LLM + embeddings | <https://platform.openai.com/api-keys> |\n",
    "| ***WaterCrawl Search or Tavily Search** | Web search for FLARE | <https://app.tavily.com/home> <https://app.watercrawl.dev/dashboard/search> |\n",
    "\n",
    "---\n",
    "\n",
    "Option 1 –keep it clean: use a `.env` file ⚠️\n",
    "\n",
    "\n",
    "Create the file **once**, store your keys, and everything else “just works”.\n",
    "\n",
    "```python\n",
    "# ── create_env.py ──\n",
    "env_text = \"\"\"\n",
    "OPENAI_API_KEY= ***put your APi key here *** \n",
    "WATERCRAWL_API_KEY=* **put your APi key here *** \n",
    "If you are using Tavily Search: TAVILY_API_KEY= ***put your APi key here *** \n",
    "\n",
    "\"\"\".strip()\n",
    "\n",
    "with open(\".env\", \"w\") as f:\n",
    "    f.write(env_text)\n",
    "print(\".env file created — now edit it with your real keys ✏️\")\n",
    "\n",
    "-------------------------------------------------\n",
    "Option 2 – quick‑and‑dirty: hard‑code in the notebook ⚠️\n",
    "\n",
    "OPENAI_API_KEY= ***put your APi key here *** \n",
    "TAVILY_API_KEY= ***put your APi key here ***  #If you are using Tavily Search:\n",
    "WATERCRAWL_API_KEY=* **put your APi key here *** \n",
    "\n",
    "Not recommended — anyone who sees or commits the notebook can read your keys.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ac82939",
   "metadata": {},
   "source": [
    "##### ➡️ **If you’re using a `.env` file load the API keys with dotenv** \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1babea5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from dotenv import load_dotenv\n",
    "import os\n",
    "\n",
    "load_dotenv()  # pulls everything from .env\n",
    "\n",
    "OPENAI_API_KEY   = os.environ.get(\"OPENAI_API_KEY\")\n",
    "WATERCRAWL_API_KEY = os.environ.get(\"WATERCRAWL_API_KEY\")\n",
    "# If you are using Tavily Search: \n",
    "TAVILY_API_KEY   = os.environ.get(\"TAVILY_API_KEY\") \n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a7e4b63d",
   "metadata": {},
   "source": [
    "##### ➡️ **Import our packages**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a7888f4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Any, List\n",
    "from langchain.callbacks.manager import (\n",
    "    AsyncCallbackManagerForRetrieverRun,\n",
    "    CallbackManagerForRetrieverRun,\n",
    ")\n",
    "from langchain_core.documents import Document\n",
    "from langchain_core.retrievers import BaseRetriever\n",
    "from watercrawl import WaterCrawlAPIClient\n",
    "from pydantic import BaseModel\n",
    "from langchain_openai import ChatOpenAI, OpenAI\n",
    "import requests\n",
    "from langchain.chains import FlareChain\n",
    "from typing import List\n",
    "from watercrawl import WaterCrawlAPIClient\n",
    "\n",
    "\n",
    "from typing import Any, List, Callable, Optional\n",
    "from pydantic import BaseModel, Field\n",
    "\n",
    "from langchain.schema import Document\n",
    "from langchain.callbacks.manager import (\n",
    "    CallbackManagerForRetrieverRun,\n",
    "    AsyncCallbackManagerForRetrieverRun,\n",
    ")\n",
    "from watercrawl import WaterCrawlAPIClient\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5f552dce",
   "metadata": {},
   "source": [
    "#### ➡️ **Lets build our search function**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b43a04ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def tavily_search_tool(\n",
    "    query: str,\n",
    "    api_key: str,\n",
    "    max_results: int = 3,\n",
    "    *,\n",
    "    topic: str = \"general\",\n",
    "    depth: str = \"basic\"          # \"basic\" | \"deep\"\n",
    ") -> List[str]:\n",
    "    \"\"\"\n",
    "    Search the web with Tavily and return a list of result URLs.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    query : str\n",
    "        The search query.\n",
    "    api_key : str\n",
    "        Your Tavily API key.\n",
    "    max_results : int, default 3\n",
    "        Maximum number of links to return.\n",
    "    topic : str, default \"general\"\n",
    "        Topical preset that adjusts result ranking.\n",
    "    depth : str, default \"basic\"\n",
    "        Search depth (\"basic\" is fastest; \"deep\" gathers more context).\n",
    "    \"\"\"\n",
    "    url = \"https://api.tavily.com/search\"\n",
    "    payload = {\n",
    "        \"query\": query,\n",
    "        \"topic\": topic,\n",
    "        \"search_depth\": depth,\n",
    "        \"max_results\": max_results,\n",
    "        \"include_answer\": False,\n",
    "        \"include_raw_content\": False,\n",
    "        \"include_domains\": [],\n",
    "        \"exclude_domains\": [],\n",
    "    }\n",
    "    headers = {\n",
    "        \"Authorization\": f\"Bearer {api_key}\",\n",
    "        \"Content-Type\": \"application/json\",\n",
    "    }\n",
    "\n",
    "    try:\n",
    "        response = requests.post(url, json=payload, headers=headers, timeout=15)\n",
    "        response.raise_for_status()\n",
    "        data = response.json()\n",
    "\n",
    "        # `data[\"results\"]` is a list of hits.\n",
    "        urls = [\n",
    "            hit[\"url\"] for hit in data.get(\"results\", []) if isinstance(hit, dict) and hit.get(\"url\")\n",
    "        ][:max_results]\n",
    "\n",
    "        print(urls)  # optional debug‑print to mirror your original helper\n",
    "        return urls\n",
    "\n",
    "    except Exception as e:\n",
    "        print(f\"⚠️  Tavily search error: {e}\")\n",
    "        return []\n",
    "\n",
    "\n",
    "def watercrawl_search_tool(\n",
    "    query: str,\n",
    "    api_key: str,\n",
    "    max_results: int = 3,\n",
    "    *,\n",
    "    language: str = \"en\",\n",
    "    country: str = \"us\",\n",
    "    time_range: str = \"month\",   # \"day\" | \"week\" | \"month\" | \"year\" | \"all\"\n",
    "    depth: str = \"basic\"         # \"basic\" | \"deep\"\n",
    ") -> List[str]:\n",
    "    \"\"\"\n",
    "    Search the web with WaterCrawl and return a list of result URLs.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    query : str\n",
    "        The search query.\n",
    "    api_key : str\n",
    "        Your WaterCrawl API key.\n",
    "    max_results : int, default 3\n",
    "        Maximum number of links to return.\n",
    "    language : str, default \"en\"\n",
    "        ISO‑639‑1 language code for the search.\n",
    "    country : str, default \"us\"\n",
    "        ISO‑3166‑1 alpha‑2 country code used to geotarget the search.\n",
    "    time_range : str, default \"month\"\n",
    "        How recent the results should be.\n",
    "    depth : str, default \"basic\"\n",
    "        Search depth (\"basic\" is fastest; \"deep\" crawls additional links).\n",
    "    \"\"\"\n",
    "    client = WaterCrawlAPIClient(api_key)\n",
    "\n",
    "    try:\n",
    "        search_request = client.create_search_request(\n",
    "            query=query,\n",
    "            search_options={\n",
    "                \"depth\": depth,\n",
    "                \"language\": language,\n",
    "                \"country\": country,\n",
    "                \"time_range\": time_range,\n",
    "                \"search_type\": \"web\",\n",
    "            },\n",
    "            result_limit=max_results,\n",
    "            sync=True,     # wait for results\n",
    "            download=True # we only need the SERP, not full pages\n",
    "        )\n",
    "\n",
    "        # `search_request[\"result\"]` is a list of hits.\n",
    "        urls = [hit[\"url\"] for hit in search_request[\"result\"] if \"url\" in hit][:max_results]\n",
    "\n",
    "        print(urls)  # optional debug‑print to mirror your original helper\n",
    "        return urls\n",
    "\n",
    "    except Exception as e:\n",
    "        print(f\"⚠️  WaterCrawl search error: {e}\")\n",
    "        return []\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9b3a698",
   "metadata": {},
   "source": [
    "#### ➡️ **Lets build WaterCrawl Retriever**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a4680b71",
   "metadata": {},
   "outputs": [],
   "source": [
    "class WaterCrawlRetriever(BaseRetriever, BaseModel):\n",
    "    \"\"\"\n",
    "    Retrieve web pages with either *WaterCrawl* or *Tavily* search, then scrape\n",
    "    the HTML via WaterCrawl for LangChain ingestion.\n",
    "\n",
    "    The default search engine is **WaterCrawl**; switch to Tavily by passing\n",
    "    ``search_tool=tavily_search_tool`` at init time.\n",
    "    \"\"\"\n",
    "\n",
    "    client: WaterCrawlAPIClient\n",
    "    tavily_api_key: Optional[str] = None\n",
    "    watercrawl_api_key: Optional[str] = None\n",
    "    search_tool: Callable[[str, str, int], List[str]] = Field(\n",
    "        default_factory=lambda: watercrawl_search_tool\n",
    "    )\n",
    "\n",
    "    # Scraper config\n",
    "    page_options: dict = {\n",
    "        \"exclude_tags\": [\"nav\", \"footer\", \"aside\"],\n",
    "        \"include_tags\": [\"article\", \"main\"],\n",
    "        \"wait_time\": 100,\n",
    "        \"include_html\": False,\n",
    "        \"only_main_content\": True,\n",
    "        \"include_links\": False,\n",
    "    }\n",
    "\n",
    "    # --------------------------------------------------------------------- #\n",
    "    # Sync                                                                   #\n",
    "    # --------------------------------------------------------------------- #\n",
    "    def _get_relevant_documents(\n",
    "        self,\n",
    "        query: str,\n",
    "        *,\n",
    "        run_manager: CallbackManagerForRetrieverRun,\n",
    "        **kwargs: Any,\n",
    "    ) -> List[Document]:\n",
    "        # Choose the right API key for the selected search function\n",
    "        if self.search_tool is tavily_search_tool:\n",
    "            api_key = self.tavily_api_key\n",
    "        else:  # default: WaterCrawl search\n",
    "            api_key = (\n",
    "                self.watercrawl_api_key\n",
    "                or getattr(self.client, \"api_key\", None)  # fallback to client key\n",
    "            )\n",
    "\n",
    "        documents: List[Document] = []\n",
    "\n",
    "        try:\n",
    "            urls = self.search_tool(query, api_key, max_results=3)\n",
    "            for url in urls:\n",
    "                try:\n",
    "                    result = self.client.scrape_url(\n",
    "                        url=url,\n",
    "                        page_options=self.page_options,\n",
    "                        sync=True,\n",
    "                        download=True,\n",
    "                    )\n",
    "                    content = result.get(\"content\", \"\")\n",
    "                    if content:\n",
    "                        documents.append(\n",
    "                            Document(page_content=content, metadata={\"source\": url})\n",
    "                        )\n",
    "                except Exception as e:  # pragma: no cover\n",
    "                    print(f\"⚠️  Scrape failed for {url}: {e}\")\n",
    "        except Exception as e:  # pragma: no cover\n",
    "            print(f\"⚠️  Search failed: {e}\")\n",
    "\n",
    "        return documents\n",
    "\n",
    "    # --------------------------------------------------------------------- #\n",
    "    # Async                                                                  #\n",
    "    # --------------------------------------------------------------------- #\n",
    "    async def _aget_relevant_documents(\n",
    "        self,\n",
    "        query: str,\n",
    "        *,\n",
    "        run_manager: AsyncCallbackManagerForRetrieverRun,\n",
    "        **kwargs: Any,\n",
    "    ) -> List[Document]:\n",
    "        # simple fall‑back to sync in worker thread\n",
    "        from asyncio import to_thread\n",
    "\n",
    "        return await to_thread(self._get_relevant_documents, query, run_manager=run_manager)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "1dac4248",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['https://github.com/langgenius/dify/releases', 'https://liduos.com/en/ai-develope-tools-series-3-open-source-ai-web-crawler-frameworks.html', 'https://github.com/langgenius/dify-plugins']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['https://github.com/langgenius/dify/releases',\n",
       " 'https://liduos.com/en/ai-develope-tools-series-3-open-source-ai-web-crawler-frameworks.html',\n",
       " 'https://github.com/langgenius/dify-plugins']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "watercrawl_search_tool(query, WATERCRAWL_API_KEY, max_results=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1070d91e",
   "metadata": {},
   "source": [
    "#### ➡️ **Create the langchain retriever obect using WaterCrawlRetriever we have built above**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "59c7d875",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "retriever = WaterCrawlRetriever(client=WaterCrawlAPIClient(api_key=WATERCRAWL_API_KEY),watercrawl_api_key=WATERCRAWL_API_KEY)\n",
    "\n",
    "# if you have tavily api key\n",
    "# retriever = WaterCrawlRetriever(client=WaterCrawlAPIClient(api_key=WATERCRAWL_API_KEY),watercrawl_api_key=WATERCRAWL_API_KEY,\n",
    "#     tavily_api_key=TAVILY_API_KEY)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "92478194",
   "metadata": {},
   "source": [
    "#### ➡️ **FLARE Chain**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "577e7c2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We set this so we can see what exactly is going on\n",
    "from langchain.globals import set_verbose\n",
    "set_verbose(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "300d783e",
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = ChatOpenAI( model=\"gpt-4o\", temperature=0)\n",
    "flare = FlareChain.from_llm(\n",
    "    llm,\n",
    "    retriever=retriever,\n",
    "    max_generation_len=164,\n",
    "    min_prob=0.3,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "1f3d5e90",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Explain what is watercrawl tool and how I can improve the LLM performance?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "4b1bfa8c",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new FlareChain chain...\u001b[0m\n",
      "\u001b[36;1m\u001b[1;3mCurrent Response: \u001b[0m\n",
      "\u001b[33;1m\u001b[1;3mGenerated Questions: ['What type of software is the Watercrawl tool?', 'What type of models does the Watercrawl tool analyze and optimize for performance?', 'How can you use the Watercrawl tool to identify areas where the LLM may be underperforming and provide suggestions for improvement?']\u001b[0m\n",
      "['https://github.com/watercrawl/watercrawl-dify-plugin/blob/main/README.md', 'https://watercrawl.dev/', 'https://www.reddit.com/r/estimators/comments/10y3a0q/best_large_commercial_estimating_software/']\n",
      "['https://www.usna.edu/AeroDept/_files/AY21_Capstone.pdf', 'https://github.com/langgenius/dify/releases', 'https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9100143/']\n",
      "⚠️  Scrape failed for https://www.usna.edu/AeroDept/_files/AY21_Capstone.pdf: 'NoneType' object has no attribute 'get'\n",
      "⚠️  Scrape failed for https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9100143/: 'NoneType' object has no attribute 'get'\n",
      "[]\n",
      "\u001b[36;1m\u001b[1;3mCurrent Response:  The Watercrawl tool is a software program used for web crawling and data extraction. It helps gather information from websites and analyze it for various purposes. To improve the LLM (Large Language Model) performance, you can optimize the data extraction process by using Watercrawl to efficiently collect relevant data that can be used to train and fine-tune the LLM model. This will ensure that the model has high-quality input data to learn from, leading to better performance. Additionally, you can also consider fine-tuning the hyperparameters of the LLM model based on the insights gained from the data extracted using Watercrawl. This can help enhance the model's accuracy and efficiency.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'user_input': 'Explain what is watercrawl tool and how I can improve the LLM performance?',\n",
       " 'response': \"The Watercrawl tool is a software program used for web crawling and data extraction. It helps gather information from websites and analyze it for various purposes. To improve the LLM (Large Language Model) performance, you can optimize the data extraction process by using Watercrawl to efficiently collect relevant data that can be used to train and fine-tune the LLM model. This will ensure that the model has high-quality input data to learn from, leading to better performance. Additionally, you can also consider fine-tuning the hyperparameters of the LLM model based on the insights gained from the data extracted using Watercrawl. This can help enhance the model's accuracy and efficiency. \"}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flare.invoke(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ➡️ **Now lets see a simple Open AI chain so we can see the value of the FLARE Chain**\n",
    "#### for the test query we provided, the answer of the same LLM is completely wrong!!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7bed8944",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\n\\nWatercrawl is a web performance testing tool that helps in analyzing the load and stress on a website or web application. It simulates real-world user traffic and measures the website's response time, throughput, and server performance under different load conditions.\\n\\nTo improve the LLM (Load, Latency, and Memory) performance using Watercrawl, the following steps can be taken:\\n\\n1. Identify bottlenecks: Watercrawl helps in identifying the areas of the website that are causing performance issues. It provides detailed reports on page load times, HTTP requests, and server response times, which can help in identifying the bottlenecks.\\n\\n2. Optimize website code: Based on the reports generated by Watercrawl, developers can optimize the website's code to reduce page load times and improve server response times. This can include techniques like minimizing HTTP requests, optimizing images, and using caching mechanisms.\\n\\n3. Test under different load conditions: Watercrawl allows testing under different load conditions, such as low, medium, and high traffic. This helps in identifying how the website performs under different levels of user traffic and if it can handle a sudden surge in traffic.\\n\\n4. Test from different geographical locations: Watercrawl offers the option to test the website's performance from different geographical locations. This helps in identifying\""
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm.invoke(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a37b8e9",
   "metadata": {},
   "source": [
    "##### 🚨⚠️ As you have noted, for the test query we provided, the answer from the **same LLM** is **completely wrong** ❌🤯‼️\n",
    "\n",
    "> 💬 It confidently gives a **wrong answer** — showing **why refinement and retrieval matter** so much in real-world usage.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "caa3a3b2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c4f0f297",
   "metadata": {},
   "source": [
    "#### ➡️ **For further information**:\n",
    "##### 📘 Introduction to FlareChain in LangChain\n",
    "\n",
    "**FlareChain** is an advanced chain in the LangChain framework 🧠⚙️ designed to *iteratively refine answers* from a language model. It improves response quality by:\n",
    "\n",
    "🔍 Identifying **low-confidence** spans  \n",
    "❓ Generating **clarifying questions**  \n",
    "📚 Retrieving **relevant context**  \n",
    "🔁 Updating the answer in a loop\n",
    "\n",
    "---\n",
    "\n",
    "#### 🧩 Key Arguments of `FlareChain`\n",
    "\n",
    "##### 🗣 2. `response_chain`\n",
    "Generates the actual response using user input + context.\n",
    "\n",
    "##### 🧾 3. `output_parser`\n",
    "Checks whether the current answer is “good enough” to stop refinement.\n",
    "\n",
    "##### 📡 4. `retriever`\n",
    "Fetches documents to provide factual backup for refining the answer.\n",
    "\n",
    "##### 📉 5. `min_prob`\n",
    "Low-confidence threshold (default: `0.2`) – tokens below this are flagged for review.\n",
    "\n",
    "##### ↔️ 6. `min_token_gap`\n",
    "Ensures separation between two flagged spans (default: `5` tokens).\n",
    "\n",
    "##### 🧷 7. `num_pad_tokens`\n",
    "Adds context tokens around flagged spans (default: `2`).\n",
    "\n",
    "##### 🔁 8. `max_iter`\n",
    "Max number of refinement cycles (default: `10`).\n",
    "\n",
    "##### 🧭 9. `start_with_retrieval`\n",
    "If `True`, starts by retrieving context even before generating the first draft.\n",
    "\n",
    "---\n",
    "\n",
    "#### 🧾 Inputs and Outputs\n",
    "\n",
    "- 📥 **Input Key**: `user_input`  \n",
    "- 📤 **Output Key**: `response`  \n",
    "\n",
    "💡 The chain processes a single user prompt and returns an *improved, confident, and context-aware response*.\n",
    "\n",
    "---\n",
    "\n",
    "#### 📚 References\n",
    "\n",
    "- [LangChain FlareChain Documentation](https://api.python.langchain.com/en/latest/langchain/chains/langchain.chains.flare.base.FlareChain.html)\n",
    "- [WaterCrawl Documentation](https://docs.watercrawl.dev/intro)\n",
    "- [WaterCrawl scrape Documentation](https://docs.watercrawl.dev/api/scrape-url)\n",
    "- [WaterCrawl search Documentation](https://docs.watercrawl.dev/api/get-search)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbadd022",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
